Active Learning in Recommender Systems.

Least%2DSquares Independent Component Analysis.

Feature Selection for Reinforcement Learning Evaluating Implicit State%2DReward Dependency via Conditional Mutual Information.

Dependence Minimizing Regression with Model Selection for Non%2DLinear Causal Inference under Non%2DGaussian Noise.

Feature Selection for Reinforcement Learning Evaluating Implicit State%2DReward Dependency via Conditional Mutual Information.

Acceleration of sequence kernel computation for real%2Dtime speaker identification.

Direct importance estimation with probabilistic principal component analyzers.

Automatic audio tagging using covariate shift adaptation.

A Fast Augmented Lagrangian Algorithm for Learning Low%2DRank Matrices.
We propose a general and efficient algorithm for learning low-rank matrices. The proposed algorithm converges super-linearly and can keep the matrix to be learned in a compact factorized representation without the need of specifying the rank beforehand. Moreover, we show that the framework can be easily generalized to the problem of learning multiple matrices and general spectral regularization. Empirically we show that we can recover a 10,000×10,000 matrix from 1.2 million observations in about 5 minutes. Furthermore, we show that in a brain-computer interface problem, the proposed method can speed-up the optimization by two orders of magnitude against the conventional projected gradient method and produces more reliable solutions. 1.
Nonparametric Return Distribution Approximation for Reinforcement Learning.
Standard Reinforcement Learning (RL) aims to optimize decision-making rules in terms of the expected return. However, especially for risk-management purposes, other criteria such as the expected shortfall are sometimes preferred. Here, we describe a method of approximating the distribution of returns, which allows us to derive various kinds of information about the returns. We first show that the Bellman equation, which is a recursive formula for the expected return, can be extended to the cumulative return distribution. Then we derive a nonparametric return distribution estimator with particle smoothingbasedonthis extendedBellman equation. A key aspect of the proposed algorithm is to represent the recursion relation in the extendedBellmanequationbyasimplereplacement procedure of particles associated with a state by using those of the successor state. We show that our algorithm leads to a risksensitive RL paradigm. The usefulness of the proposed approach is demonstrated through numerical experiments. 1.
Implicit Regularization in Variational Bayesian Matrix Factorization.
Matrix factorization into the product of lowrank matrices induces non-identifiability, i.e., the mapping between the target matrix and factorized matrices is not one-to-one. In this paper, we theoretically investigate the influence of non-identifiability on Bayesian matrix factorization. More specifically, we show that a variational Bayesian method involves regularization effect even when the prior is non-informative, which is intrinsically different from the maximum a posteriori approach. We also extend our analysis to empirical Bayes scenarios where hyperparameters are also learned from data. 1.
SemiCCA Efficient Semi%2Dsupervised Learning of Canonical Correlations.

Perceived Age Estimation under Lighting Condition Change by Covariate Shift Adaptation.

Direct Density Ratio Estimation with Dimensionality Reduction.

Direct Density Ratio Estimation with Dimensionality Reduction.

Semi%2Dsupervised Estimation of Perceived Age from Face Images.

Multi%2Dparametric Solution%2Dpath Algorithm for Instance%2Dweighted Support Vector Machines

Conic Programming for Multitask Learning.

Theoretical Analysis of Density Ratio Estimation.

Foreword.
reference set of international guidelines and recommendations for the reporting and presentation of statistical data and metadata. Although, the Handbook was prepared primarily for short-term economic statistics, many, if not most of the recommendations presented are also relevant for annual (structural) statistics and for social and population statistics. The Handbook draws heavily on existing international statistical guidelines and recommendations that have been developed over the last twenty years by international organisations and national statistical agencies. These international standards have been developed for implementation by international organisations themselves and by national agencies in both developed and developing economies. The need for the articulation of a comprehensive set of recommendations on the reporting and presentation of statistical data and metadata is one aspect of the overall demand for improved international comparability and consistency of statistics compiled and published by national agencies and international organisations. This requirement has itself been driven by closer economic and social linkages between both the 30 OECD Member countries and between those countries and large emerging non-member economies. As discussed in the Handbook, the major causes of differences in statistics compiled in different countries flow from the use of different definitions and concepts and

Superfast%2DTrainable Multi%2DClass Probabilistic Classifier by Least%2DSquares Posterior Fitting.


A Semi%2DSupervised Approach to Perceived Age Prediction from Face Images.

Least%2DSquares Conditional Density Estimation.

Spectral Methods for Thesaurus Construction.

Least Absolute Policy Iteration%2D%2DA Robust Approach to Value Function Approximation.

Semi%2Dsupervised local Fisher discriminant analysis for dimensionality reduction.
Abstract—Canonical correlation analysis (CCA) is a powerful tool for analyzing multi-dimensional paired data. However, CCA tends to perform poorly when the number of paired samples is limited, which is often the case in practice. To cope with this problem, we propose a semi-supervised variant of CCA named “semiCCA”that allows us to incorporate additional unpaired samples for mitigating over ttng. The proposed method smoothly bridges the eigenvalue problems of CCA and principal component analysis (PCA), and thus its solution can be computed ef ciently just by solving a single eigenvalue problem as the original CCA. Index Terms—Canonical correlation analysis, semi-supervised learning, generalized eigenproblem, automatic image annotation I.
Dimensionality reduction for density ratio estimation in high%2Ddimensional spaces.

Efficient exploration through active learning for value function approximation in reinforcement learning.
Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. We demonstrate the usefulness of the proposed method, named active policy iteration (API), through simulations with a batting robot. 1
Efficient exploration through active learning for value function approximation in reinforcement learning.
Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. We demonstrate the usefulness of the proposed method, named active policy iteration (API), through simulations with a batting robot. 1
Semi%2Dsupervised speaker identification under covariate shift.

Density Ratio Estimation A New Versatile Tool for Machine Learning.

A Framework of Adaptive Brain Computer Interfaces.

Efficient Sample Reuse in EM%2DBased Policy Search.
Abstract. Direct policy search is a promising reinforcement learning framework in particular for controlling in continuous, high-dimensional systems such as anthropomorphic robots. Policy search often requires a large number of samples for obtaining a stable policy update estimator due to its high flexibility. However, this is prohibitive when the sampling cost is expensive. In this paper, we extend an EM-based policy search method so that previously collected samples can be efficiently reused. The usefulness of the proposed method, called Reward-weighted Regression with sample Reuse (R 3), is demonstrated through a robot learning experiment. 1
Estimating Squared%2DLoss Mutual Information for Independent Component Analysis.

Covariate shift adaptation for semi%2Dsupervised speaker identification.

Least absolute policy iteration for robust value function approximation.

Active Policy Iteration Efficient Exploration through Active Learning for Value Function Approximation in Reinforcement Learning.
Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. We demonstrate the usefulness of the proposed method, named active policy iteration (API), through simulations with a batting robot. 1
Probabilistic principal component analysis based on JoyStick Probability Selector.

Analysis of Variational Bayesian Matrix Factorization.

Link Propagation A Fast Semi%2Dsupervised Learning Algorithm for Link Prediction.
We propose Link Propagation as a new semi-supervised learning method for link prediction problems, where the task is to predict unknown parts of the network structure by using auxiliary information such as node similarities. Since the proposed method can fill in missing parts of tensors, it is applicable to multi-relational domains, allowing us to handle multiple types of links simultaneously. We also give a novel efficient algorithm for Link Propagation based on an accelerated conjugate gradient method.
Change%2DPoint Detection in Time%2DSeries Data by Direct Density%2DRatio Estimation.

Mutual information estimation reveals global associations between stimuli and biological processes.

Simultaneous inference of biological networks of multiple species from genome%2Dwide data and evolutionary information a semi%2Dsupervised approach.

Efficient Construction of Neighborhood Graphs by the Multiple Sorting Method

Robust Label Propagation on Multiple Networks.
Transductive inference on graphs such as label propagation algorithms is receiving a lot of attention. In this paper, we address a label propagation problem on multiple networks and present a new algorithm that automatically integrates structure information brought in by multiple networks. The proposed method is robust in that irrelevant networks are automatically deemphasized, which is an advantage over Tsuda et al.’s approach [14]. We also show that the proposed algorithm can be interpreted as an EM algorithm with a Student-t prior. Finally, we demonstrate the usefulness of our method in protein function prediction.
Direct Importance Estimation with Gaussian Mixture Models.

On Computational Issues of Semi%2DSupervised Local Fisher Discriminant Analysis.

Recent Advances and Trends in Large%2DScale Kernel Methods.

A Least%2Dsquares Approach to Direct Importance Estimation.

Pool%2Dbased active learning in approximate linear regression.
The goal of pool-based active learning is to choose the best input points to gather output values from a ‘pool ’ of input samples. We develop two pool-based active learning criteria for linear regression. The first criterion allows us to obtain a closedform solution so it is computationally very efficient. However, this solution is not necessarily optimal in the single-trial generalization error analysis. The second criterion can give a better solution, but it does not have a closed-form solution and therefore some additional search strategy is needed. To cope with this problem, we propose a practical procedure which enables us to efficiently search for a better solution around the optimal solution of the first method. Simulations with toy and benchmark datasets show that the proposed active learning method compares favorably with other active learning methods as well as the baseline passive learning scheme. Furthermore, the usefulness of the proposed active learning method is also demonstrated in wafer alignment in semiconductor exposure apparatus.
On Generalization Performance and Non%2DConvex Optimization of Extended nu%2DSupport Vector Machine.

Theory and Algorithm for Learning with Dissimilarity Functions.

Adaptive importance sampling for value function approximation in off%2Dpolicy reinforcement learning.
Off-policy reinforcement learning is aimed at efficiently using data samples gathered from a policy that is different from the currently optimized policy. A common approach is to use importance sampling techniques for compensating for the bias of value function estimators caused by the difference between the data-sampling policy and the target policy. However, existing off-policy methods often do not take the variance of the value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations. 1
On Generalization Performance and Non%2DConvex Optimization of Extended nu%2DSupport Vector Machine.

Adaptive Importance Sampling with Automatic Model Selection in Value Function Approximation.
Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past, which is an essential problem for physically grounded AI as experiments are usually prohibitively expensive. A common approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-sampling policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations.
On the Margin Explanation of Boosting Algorithms.
Much attention has been paid to the theoretical explanation of the empirical success of AdaBoost. The most influential work is the margin theory, which is essentially an upper bound for the generalization error of any voting classifier in terms of the margin distribution over the training data. However, Breiman raised important questions about the margin explanation by developing a boosting algorithm arc-gv that provably generates a larger minimum margin than AdaBoost. He also gave a sharper bound in terms of the minimum margin, and argued that the minimum margin governs the generalization. In experiments however, arc-gv usually performs worse than AdaBoost, putting the margin explanation into serious doubts. In this paper, we try to give a complete answer to Breiman’s critique by proving a bound in terms of a new margin measure called Equilibrium margin (Emargin). The Emargin bound is uniformly sharper than Breiman’s minimum margin bound. This result suggests that the minimum margin is not crucial for the generalization error. We also show that a large Emargin implies good generalization. Experimental results on benchmark datasets demonstrate that AdaBoost usually has a larger Emargin and a smaller test error than arc-gv, which agrees well with our theory. 1
Pool%2DBased Agnostic Experiment Design in Linear Regression.

nu%2Dsupport vector machine as conditional value%2Dat%2Drisk minimization.

Inlier%2DBased Outlier Detection via Direct Density Ratio Estimation.

nu%2Dsupport vector machine as conditional value%2Dat%2Drisk minimization.

Order Retrieval.
Collaborative filters help people make choices based on the opinions of other people. GroupLens is a system for collaborative filtering of netnews, to help people find articles they will like in the huge stream of available articles. News reader clients display predicted scores and make it easy for users to rate articles after they read them. Rating servers, called Better Bit Bureaus, gather and disseminate the ratings. The rating servers predict scores based on the heuristic that people who agreed in the past will probably agree again. Users can protect their privacy by entering ratings under pseudonyms, without reducing the effectiveness of the score prediction. The entire architecture is open: alternative software for news clients and Better Bit Bureaus can be developed independently and can interoperate with the components we have developed.
Order Retrieval.
Collaborative filters help people make choices based on the opinions of other people. GroupLens is a system for collaborative filtering of netnews, to help people find articles they will like in the huge stream of available articles. News reader clients display predicted scores and make it easy for users to rate articles after they read them. Rating servers, called Better Bit Bureaus, gather and disseminate the ratings. The rating servers predict scores based on the heuristic that people who agreed in the past will probably agree again. Users can protect their privacy by entering ratings under pseudonyms, without reducing the effectiveness of the score prediction. The entire architecture is open: alternative software for news clients and Better Bit Bureaus can be developed independently and can interoperate with the components we have developed.
Efficient Direct Density Ratio Estimation for Non%2Dstationarity Adaptation and Outlier Detection.

Semi%2DSupervised Local Fisher Discriminant Analysis for Dimensionality Reduction.
Abstract—Canonical correlation analysis (CCA) is a powerful tool for analyzing multi-dimensional paired data. However, CCA tends to perform poorly when the number of paired samples is limited, which is often the case in practice. To cope with this problem, we propose a semi-supervised variant of CCA named “semiCCA”that allows us to incorporate additional unpaired samples for mitigating over ttng. The proposed method smoothly bridges the eigenvalue problems of CCA and principal component analysis (PCA), and thus its solution can be computed ef ciently just by solving a single eigenvalue problem as the original CCA. Index Terms—Canonical correlation analysis, semi-supervised learning, generalized eigenproblem, automatic image annotation I.
Direct Density Ratio Estimation for Large%2Dscale Covariate Shift Adaptation.
Covariate shift is a situation in supervised learning where training and test inputs follow different distributions even though the functional relation remains unchanged. A common approach to compensating for the bias caused by covariate shift is to reweight the training samples according to importance, which is the ratio of test and training densities. We propose a novel method that allows us to directly estimate the importance from samples without going through the hard task of density estimation. An advantage of the proposed method is that the computation time is nearly independent of the number of test input samples, which is highly beneficial in recent applications with large numbers of unlabeled samples. We demonstrate through experiments that the proposed method is computationally more efficient than existing approaches with comparable accuracy.
Active Learning with Model Selection in Linear Regression.
Optimally designing the location of training input points (active learning) and choosing the best model (model selection) are two important components of supervised learning and have been studied extensively. However, these two issues seem to have been investigated separately as two independent problems. If training input points and models are simultaneously optimized, the generalization performance would be further improved. In this paper, we propose a new approach called ensemble active learning for solving the problems of active learning and model selection at the same time. We demonstrate by numerical experiments that the proposed method compares favorably with alternative approaches such as iteratively performing active learning and model selection in a sequential manner. 
Integration of Multiple Networks for Robust Label Propagation.
Transductive inference on graphs such as label propagation algorithms is receiving a lot of attention. In this paper, we address a label propagation problem on multiple networks and present a new algorithm that automatically integrates structure information brought in by multiple networks. The proposed method is robust in that irrelevant networks are automatically deemphasized, which is an advantage over Tsuda et al.’s approach [14]. We also show that the proposed algorithm can be interpreted as an EM algorithm with a Student-t prior. Finally, we demonstrate the usefulness of our method in protein function prediction.
Geodesic Gaussian kernels for value function approximation.
Abstract The least-squares policy iteration approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in simulated robot arm control and Khepera robot navigation. The current paper is a complete version of our earlier manuscript (Sugiyama et al. 2007). The major differences are that we included more technical details of the proposed method in Sect. 3, discussions on the relation to related methods in Sect. 4, and the application to map building in Sect. 6. A demo movie of the proposed method applied in simulated robot arm control and Khepera robot navigation is available from
Approximating the Best Linear Unbiased Estimator of Non%2DGaussian Signals with Gaussian Noise.

A batch ensemble approach to active learning with model selection.

Asymptotic Bayesian generalization error when training and test distributions are different.
In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian generalization error and establish its upper bound under a very general setting. Our important finding is that lower order terms—which can be ignored in the absence of the distribution change— play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.
Value Function Approximation on Non%2DLinear Manifolds for Robot Motor Control.
Abstract — The least squares approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in realworld reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in a simulated robot arm control and Khepera robot navigation. I.
Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation.

Multi%2DTask Learning via Conic Programming.
When we have several related tasks, solving them simultaneously is shown to be more effective than solving them individually. This approach is called multi-task learning (MTL) and has been studied extensively. Existing approaches to MTL often treat all the tasks as uniformly related to each other and the relatedness of the tasks is controlled globally. For this reason, the existing methods can lead to undesired solutions when some tasks are not highly related to each other, and some pairs of related tasks can have significantly different solutions. In this paper, we propose a novel MTL algorithm that can overcome these problems. Our method makes use of a task network, which describes the relation structure among tasks. This allows us to deal with intricate relation structures in a systematic way. Furthermore, we control the relatedness of the tasks locally, so all pairs of related tasks are guaranteed to have similar solutions. We apply the above idea to support vector machines (SVMs) and show that the optimization problem can be cast as a second order cone program, which is convex and can be solved efficiently. The usefulness of our approach is demonstrated through simulations with protein super-family classification and ordinal regression problems. 1
Influence%2Dbased collaborative active learning.

Analytic Optimization of Adaptive Ridge Parameters Based on Regularized Subspace Information Criterion.

Generalization Error Estimation for Non%2Dlinear Learning Methods.

A New Meta%2DCriterion for Regularized Subspace Information Criterion.

Importance%2DWeighted Cross%2DValidation for Covariate Shift.

A Novel Dimension Reduction Procedure for Searching Non%2DGaussian Subspaces.

Local Fisher discriminant analysis for supervised dimensionality reduction.
Dimensionality reduction is one of the important preprocessing steps in high-dimensional data analysis. In this paper, we consider the supervised dimensionality reduction problem where samples are accompanied with class labels. Traditional Fisher discriminant analysis is a popular and powerful method for this purpose. However, it tends to give undesired results if samples in some class form several separate clusters, i.e., multimodal. In this paper, we propose a new dimensionality reduction method called local Fisher discriminant analysis (LFDA), which is a localized variant of Fisher discriminant analysis. LFDA takes local structure of the data into account so the multimodal data can be embedded appropriately. We also show that LFDA can be extended to non-linear dimensionality reduction scenarios by the kernel trick. 1.
Mixture Regression for Covariate Shift.

Model Selection Using a Class of Kernels with an Invariant Metric.

Analytic Optimization of Shrinkage Parameters Based on Regularized Subspace Information Criterion.

Constructing Kernel Functions for Binary Regression.

Active Learning in Approximately Linear Regression Based on Conditional Expectation of Generalization Error.
The goal of active learning is to determine the locations of training input points so that the generalization  error is minimized. We discuss the problem of active learning in linear regression scenarios. Traditional active
In Search of Non%2DGaussian Components of a High%2DDimensional Distribution.
This research was supported by the Deutsche Forschungsgemeinschaft through the SFB 649 &amp;quot;Economic Risk&amp;quot;.
Model Selection Under Covariate Shift.
Abstract. A common assumption in supervised learning is that the training and test input points follow the same probability distribution. However, this assumption is not fulfilled, e.g., in interpolation, extrapolation, or active learning scenarios. The violation of this assumption— known as the covariate shift—causes a heavy bias in standard generalization error estimation schemes such as cross-validation and thus they result in poor model selection. In this paper, we therefore propose an alternative estimator of the generalization error. Under covariate shift, the proposed generalization error estimator is unbiased if the learning target function is included in the model at hand and it is asymptotically unbiased in general. Experimental results show that model selection with the proposed generalization error estimator is compared favorably to crossvalidation in extrapolation. 1
Active Learning for Misspecified Models.
Active learning is the problem in supervised learning to design the locations  of training input points so that the generalization error is minimized. Existing active learning
Non%2DGaussian Component Analysis a Semi%2Dparametric Framework for Linear Dimension Reduction.
We propose a new linear method for dimension reduction to identify non-Gaussian components in high dimensional data. Our method, NGCA (non-Gaussian component analysis), uses a very general semi-parametric framework. In contrast to existing projection methods we define what is uninteresting (Gaussian): by projecting out uninterestingness, we can estimate the relevant non-Gaussian subspace. We show that the estimation error of finding the non-Gaussian components tends to zero at a parametric rate. Once NGCA components are identified and extracted, various tasks can be applied in the data analysis process, like data visualization, clustering, denoising or classification. A numerical study demonstrates the usefulness of our method. 1
Regularizing generalization error estimators a novel approach to robust model selection.

Estimating the error at given test input points for linear regression.

Trading Variance Reduction with Unbiasedness The Regularized Subspace Information Criterion for Robust Model Selection in Kernel Regression.
A well-known result by Stein (1956) shows that in particular situations, biased estimators  can yield better parameter estimates than their generally preferred unbiased  counterparts. This paper follows the same spirit as we will stabilize the unbiased  generalization error estimates by regularization and finally obtain more robust  model selection criteria for learning. We trade a small bias against a larger variance  reduction which has the beneficial e#ect of being more precise on a single training  set. We focus on the subspace information criterion (SIC), which is an unbiased  estimator of the expected generalization error measured by the reproducing kernel  Hilbert space norm. SIC can be applied to the kernel regression and it was shown  in earlier experiments that a small regularization of SIC has a stabilization e#ect.
Selecting Ridge Parameters in Infinite Dimensional Hypothesis Spaces.

The Subspace Information Criterion for Infinite Dimensional Hypothesis Spaces.
A central problem in learning is selection of an appropriate model. This is typically done  by estimating the unknown generalization errors of a set of models to be selected from and  then choosing the model with minimal generalization error estimate. In this article, we  discuss the problem of model selection and generalization error estimation in the context of  kernel regression models, e.g., kernel ridge regression, kernel subset regression or Gaussian  process regression.
Theoretical and Experimental Evaluation of the Subspace Information Criterion.
Recently, a new model selection criterion called the subspace information criterion (SIC) was proposed. SIC works well with small samples since it gives an unbiased estimate of the generalization error with finite samples. In this paper, we theoretically and experimentally evaluate the e#ectiveness of SIC in comparison with existing model selection techniques including the traditional leave-one-out cross-validation (CV), Mallows&#039;s C P , Akaike&#039;s information criterion (AIC), Sugiura&#039;s corrected AIC (cAIC), Schwarz&#039;s Bayesian information criterion (BIC), Rissanen&#039;s minimum description length criterion (MDL), and Vapnik&#039;s measure (VM). Theoretical evaluation includes the comparison of the generalization measure, approximation method, and restriction on model candidates and learning methods. Experimentally, the performance of SIC in various situations is investigated. The simulations show that SIC outperforms existing techniques especially when the number of training examples is small and the noise variance is large.  Keywords  supervised learning, generalization capability, model selection, subspace information criterion, small samples  Theoretical and Experimental Evaluation of Subspace Information Criterion 2 1 
Optimal design of regularization term and regularization parameter by subspace information criterion.
The problem of designing the regularization term and regularization parameter for linear regression models is discussed. Previously, we derived an approximation to the generalization error called the subspace information criterion (SIC), which is an unbiased estimator of the generalization error with finite samples under certain conditions. In this paper, we apply SIC to regularization learning and use it for (a) choosing the optimal regularization term and regularization parameter from given candidates, and (b) obtaining the closed form of the optimal regularization parameter for a fixed regularization term. The effectiveness of SIC is demonstrated through computer simulations with artificial and real data. Keywords supervised learning, generalization error, linear regression, regularization learning, ridge regression, model selection, regularization parameter, subspace information criterion Optimal Regularization by SIC 2 Nomenclature f(x) : learning target function D: domain of f(x) xm: m-th sample point ym: m-th sample value ɛm: m-th noise (xm,ym) : m-th training example M: the number of training examples y: M-dimensional vector consisting of {ym} M m=1 ɛ: M-dimensional vector consisting of {ɛm} M m=1 ϕp(x) : p-th basis function θp: p-th coefficient µ: the number of basis functions JG: generalization error JTE: training error JR: regularized training error T: regularization matrix α: regularization parameter A: design matrix XT,α: regularization learning matrix U: µ-dimensional matrix θ: true parameter ˆθT,α: regularization estimate ˆθu: unbiased estimate σ 2: noise variance 1
A unified method for optimizing linear image restoration filters.
Image restoration from degraded images lies at the foundation of image processing,  pattern recognition, and computer vision, so it has been extensively studied. A  large number of image restoration filters have been devised so far. It is known that  a certain filter works excellently for a certain type of original image or degradation.
Incremental Active Learning for Optimal Generalization.
The problem of designing input signals for optimal generalization is called active learning. In this paper, we give a two-stage sampling scheme for reducing both the bias and variance, and based on this scheme, we propose two active learning methods. One is the multi-point-search method applicable to arbitrary models. The effectiveness of this method is shown through computer simulations. The other is the optimal sampling method in trigonometric polynomial models. This method precisely specifies the optimal sampling locations.
Subspace Information Criterion for Model Selection.
The problem of model selection is considerably important for acquiring higher levels of generalization capability in supervised learning. In this paper, we propose a new criterion for model selection called the subspace information criterion (SIC), which is a generalization of Mallows ’ C L. It is assumed that the learning target function belongs to a specified functional Hilbert space and the generalization error is defined as the Hilbert space squared norm of the difference between the learning result function and target function. SIC gives an unbiased estimate of the generalization error so defined. SIC assumes the availability of an unbiased estimate of the target function and the noise covariance matrix, which are generally unknown. A practical calculation method of SIC for least mean squares learning is provided under the assumption that the dimension of the Hilbert space is less than the number of training examples. Finally, computer simulations in two examples show that SIC works well even when the number of training examples is small.
Incremental projection learning for optimal generalization.
In many practical situations in supervised learning, it is often expected to further improve the generalization capability after the learning process has been completed. One of the common approaches to improving the generalization capability is to add training examples. In view of the learning methods of human beings, it seems natural to build posterior learning results upon prior results, which is generally referred to as incremental learning. In this paper, a method of incremental projection learning (IPL) is presented. IPL provides exactly the same learning result as that obtained by batch projection learning. The effectiveness of the presented method is demonstrated through computer simulations.
Properties of incremental projection learning.
We proposed a method of incremental projection learning which provides exactly the same generalization capability as that obtained by batch projection learning in the previous paper. However, properties of the method have not yet been investigated. In this paper, we analyze its properties from the following aspects: First, it is shown that some of the training data which is regarded as redundant in most incremental learning methods have potential effectiveness, i.e. they will contribute to better generalization capability in the future learning process. Based on this fact, an improved criterion for redundancy of additional training data is derived. Second, the relationship between prior and posterior learning results is investigated where effective training data is classi®ed into two categories from the viewpoint of improving generalization capability. Finally, a simpler representation of incremental projection learning under certain conditions is given. The size of memory required
A new information criterion for the selection of subspace models.

Incremental Active Learning with Bias Reduction.

Training Data Selection for Optimal Generalization in Trigonometric Polynomial Networks.
In this paper, we consider the problem of active learning in trigonometric polynomial networks and give a necessary and su#cient condition of sample points to provide the optimal generalization capability. By analyzing the condition from the functional analytic point of view, we clarify the mechanism of achieving the optimal generalization capability. We also show that a set of training examples satisfying the condition does not only provide the optimal generalization but also reduces the computational complexity and memory required for the calculation of learning results. Finally, examples of sample points satisfying the condition are given and computer simulations are performed to demonstrate the e#ectiveness of the proposed active learning method. 1 Introduction  Supervised learning is obtaining an underlying rule from training examples, and can be formulated as a function approximation problem. If sample points are actively designed, then learning can be performed more e#ciently. I...
